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Abstract 

We prove that the mapping class group of a closed oriented surface of genus p > 3 
has no proper subgroup of index < Ap + 4. 

AMS Subject Classification. Primary: 57M99. Secondary: 20F36. 



Let E be a closed oriented surface of genus p > 1, and let Homeo + (S) be the group of 
homeomorphisms h : X — > X that preserve the orientation. The mapping class group 
of S is defined to be the group .M(£) = 7r (Homeo + (E)) of isotopy classes of elements 
of Homeo + (S). This group plays an important role in low- dimensional topology, in Te- 
ichmuller theory, and in group theory. We refer to [6] for a general survey on the subject, 
and turn to the main object of the present paper: the finite index subgroups of ,M(£). 

A group G is called residually finite if, for all g £ G \ {1}, there exists a homomorphism 
ip : G — > if such that is finite and ip(g) ^ 1. By pE], the mapping class group .M(E) 
is residually finite. This means that A4(E) has many finite index subgroups, numerous 
enough to separate its elements. Nevertheless, we are going to prove: 

Theorem 1. Let £ be a closed oriented surface of genus p > 3. Then A^(S) /ias no 
proper subgroup of index < 4p + 4. 

Remark. If p — 1, then ifi(.M(£)) ~ Z/12Z, thus .M(E) has (normal) subgroups of 
index 2 and 3. If p = 2, then # X (.M(£)) ~ Z/10Z (see [9]), thus Af(E) has (normal) 
subgroups of index 2 and 5. So, the hypothesis p > 3 is needed in Theorem 1. Note also 
that the statement is valid for all the subgroups, not only for the normal ones. 
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The rest of the article is dedicated to the proof of Theorem 1. So, from now on, S denotes 
a closed oriented surface of genus p > 3. 

The two main tools that we use in our proof are the braid groups and the Dehn twists. 
For n > 2, we denote by B n the braid group on n strands. This has a presentation with 
generators a\, . . . , <r n _i and relations 

crjCTj = (Xj<7j if \i — j\ > 2 , 
CTj(Tj(Ti = OjOiOj if \i — j\ = 1 . 

The Dehn twist along an essential circle a : S 1 S will be denoted by r a . We assume 
the reader to be familiar with these two concepts, and we refer to [2] for a detailed text 
on them. 

The following is easy to prove and can be found, for instance, in [3]. 

Lemma 2. Let a, b be two essential circles which intersect transversely. Then 

T a n = nr a if a n b = , 
T a T b T a = T b r a T b if\anb\ = l. 



Now, consider the essential circles a ,ai, . . . } a 2p+ i drawn in Figure 1, and, for < % < 
2p + 1, let Tj denote the Dehn twist along a^. 




Figure 1. Essential circles in E. 



Proposition 3. 

1. The mapping <Ji i— > T{, 1 < % < 2p + l, induces a homomorphism i : B 2p +2 —* -M(E). 
Furthermore, we have the relations 

To r i — T i T o for 1 < i < 2p + 1 and i ^ 4 , 

T0 T 4T"0 — T 4 T 0' r 4 • 

2. (Humphries [H]) The Dehn twists r , r 1; . . . , r 2p generate M.(H). 

3. (Powell pn]) Hx(M(T,)) = {0}. 
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Remark. Proposition 3.1 is a straightforward consequence of Lemma 2. Note also that 
1 '■ &2 P +2 — > .M(£) is not injective (see [7j). 

For k > 1, we denote by Sym fc the symmetric group of {1, 2, ... , k}. Let G be a group. A 
homomorphism tp : G — > Sym fc is called transitive if Im <p acts transitively on {1, 2, . . . , k}. 
If : G — ► Sym fe is a transitive homomorphism, then Stab v ,(l) = {g G Gr;<p(#)(l) = 1} 
is an index k subgroup of G. Reciprocally, if H is an index k subgroup of G, then there 
exists a transitive homomorphism tp : G — > Sym fc such that H = Stab v (l) (take the action 
of G on the left cosets of H). So, in order to prove Theorem 1, it suffices to prove that 
there is no transitive homomorphism p : A4(H) — > Sym fc for any 2 < fc < 2p + 4. 

For u G Syrrik, we denote by Ad u : Sym fe — > Sym fc the inner automorphism which sends 
v to uvu^ 1 for all v G Sym fc . We say that two homomorphisms ip,ip f : G —> Sym fc 
are conjugate if there exists u G Sym fc such that ip' = Ad u op. To be conjugate is an 
equivalence relation that we denote by p ~ p'. The product of two homomorphisms 
Pi : G — > Sym fci and p> 2 '■ G — > Sym fca is the homomorphism p 1 x <p 2 '■ G — > Sym fci+fca 
defined by 



Clearly, for every homomorphism p : G — ► Sym fc , there exists a partition k 2 , . . . , k p ) 



Moreover, this decomposition is unique up to a permutation of the factors. 

Using the above defined product of homomorphisms, it is easily shown that the existence 
of a transitive homomorphism ip : .M(£) — > Sym fc for some 2 < < 4p + 4 is equivalent 
to the existence of a non-trivial homomorphism p : — > Sym 4p+4 . 

We set n = 2p+2 and we turn now to give the classification of the homomorphisms from B n 
to Sym 2n (see Corollary 7). Then, given a homomorphism p : M.(E) — > Sym 2n = Sym 4p+4 , 
our strategy for proving that p is trivial will consist on exploring the different possibilities 
for the composition ip = p o t : B„ , — > Sym 2n , where i : B n — B 2p +2 — * A^(S) is the 
homomorphism of Proposition 3.1. 

Let k G N and let w G Sym fc . Then the mapping i— > w, 1 < i < n — 1, induces 
a homomorphism cycl„, : i3 n — > Sym fc called a cyclic homomorphism. Note that cycl^ 
is transitive if and only if w is a cycle of length /c. On the other hand, the mapping 
Oi i — > G Sym n , 1 < i < n — 1, induces a transitive homomorphism p s : B n Sym n 

called the standard homomorphism. 

Theorem 4 (Artin [lj). Lei n > 7. 



(<Pi x ^2)(^)(i) = 



p 2 {g){i - h) + h 



if 1 < % < k , 

if ki + 1 < % < ki + k 2 . 
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1. For k < n, every transitive homomorphism ip : B n — > Sym fc is cyclic. 

2. Every transitive homomorphism tp : B n — > Sym n is either cyclic or conjugate to the 
standard homomorphism ps- 

Remark. For n = 4 or 6, there are transitive homomorphisms B n — > Sym n which are 
neither cyclic, nor conjugate to p$'- there are 3 for n = 4, and 1 for n = 6, up to 
conjugation (see flj). However, in this paper we are concerned only by the case n > 8. 

Proposition 5 (Lin |8J). Let n > 7. 

1. The mapping 

Oi i— > (i, n + i + 1, n + i, i + 1) , 1 < i < n — 1 , 
induces a transitive homomorphism pn : B n — > Sym 2n . 

2. The mapping 

i—l n 

a i ^ Jj0'» n+ j) ■ (i,n + i + l,n + i,i + l) ■ (j> + j) , l<i<n-l, 

j=l j=i+2 

induces a transitive homomorphism p^ 2 : B n — > Sym 2n . 

3. The mapping 

i—l n 

o% >-> Yl(j, n + j) ■ (z, n + i + l)(z + 1, n + i) ■ \ [ (./. // + ./) , 1 < i < n - 1 , 



induces a transitive homomorphism pL% : £> n — > Sym. 
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The homomorphisms Pli, PL2, Pi3 of Proposition 5 will be called Lin homomorphisms. 
Theorem 6 (Lin [8J). Let n > 7. 

1. For n + l<k<2n — 1, every transitive homomorphism ip : B n — > Sym fe is cyclic. 

2. Every transitive homomorphism ip : B n — > Sym 2n is either cyclic or conjugate to 
some pLj, where j G {1,2,3}. 

Corollary 7. Let n > 7. Let ip : B n — ► Sym 2n 6e a homomorphism. Then either 

(a) ip is cyclic; or 

(b) ij) ~ ps x cycl^ /or some u> G Sym n ; or 



(c) ip ~ p s x p s ; or 

(d) ip ~ p Lj for some j G {1,2,3}. 



Proof of Theorem 1. Set n = 2p + 2 > 8. Let ip : M(H) -> Sym 4p+4 = Sym 2n 
be a homomorphism. As pointed out before, we must prove that tp is trivial. We set 
ip = ip o t : B n — > Sym 2n , where i :B n ^> Mill) is the homomorphism of Proposition 3.1. 
By Corollary 7, we have either 

(a) V — c y ci «, f° r some w G Sym 2n ; or 

(b) ip ~ p5 x cycl w for some u> G Sym n ; or 

(c) ip ~ p 5 x p 5 ; or 

(d) ^ ~ pLj for some j G {1, 2, 3}. 

Case 1: Suppose ip = cycl^, where w G Sym 2n . 

Set / = r 4 ro and observe that /r 4 / _1 = r , and frif^ 1 = T\. We have <p( T i) = = w 

for all 1 < i < 2p. Moreover, 

¥>0o) = ^(/M^M/ -1 ) = vU'MnMr 1 ) = <p(n) = w. 

Since A4(H) is generated by r , ri, . . . , r 2p (see Proposition 3.2), it follows that the image 
of <p is the cyclic group generated by w: an abelian group. Since Hi(Ai(H)) = {0} (see 
Proposition 3.3), we conclude that Imip is trivial, that is, w — 1 and <p is trivial. 

Case 2: Suppose ip = ps x cycl w , where u> G Sym n . 

For t> G Sym 2n , we define the support of v to be Supp(t>) = {i G {1, . . . , 2n}; v(i) ^ i}. 
We will often use the fact that two elements commute if their supports are disjoint. 

Let w G Sym 2n be defined by 

/ \ _ / i if 1 < i < n , 

Wo ^ w(i — n) + n if n + 1 < ? < 2n . 

So, Supp(w ) C {n + 1, n + 2, . . . , 2n}, and 

<p(Tj) = ^(^i) — (h i + l) w o ) for all 1 < i < n — 1 . 

Set m — < ^(' r o)- The permutation tt commutes with (p(r!)(p(r 2 ) _1 = (1,2,3), with 
pfaMrs)- 1 = (2,3,4), and with y^n)^)- 1 = (1,2)(3,4), thus 

«o({l,2,3}) = {1,2,3}, m ({2,3,4}) = {2,3,4}, 
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and either 

«o({1,2}) = {1,2} and u ({3, 4}) = {3, 4} , or 
uo({l,2}) = {3,4} and u ({3, 4}) = {1, 2} . 

This clearly implies that uo(i) — % for all 1 < % < 4. Similarly, the fact that uq commutes 
with v ? ( r i) — + l)^o fo r all 5 < i < n — 1 implies that uo(i) = % for all 5 < % < n. So, 
Supp(-u ) C {n + 1, . . . , 2n}. Now, from the equality tqt^tq = t^tqt^ follows 

(4, 5)u w u = (4, 5) (4, 5)w u w = w u w , 

and this last equality cannot hold: a contradiction. 

Case 3: Suppose ip = ps x ps- 

We have 

</?(rj) = ip((Ji) — (i, i + l)(n + i, n + i + 1) , for all 1 < i < n — 1 . 

Set Mo — i p{ T o)- Recall that r = /r 4 / _1 , where / = T 4 r , thus % is conjugate to 
y?(T 4 ) = (4, 5)(n + 4, n + 5), therefore wo is the disjoint product of two transpositions. 
Since uq commutes with (pfa) = (i, i + l)(n + i, n + % + 1) for all 1 < i < 3, we have 

«o({M + + + i + 1}) = {i, % + 1, n + i, n + i + 1} 

for all 1 < i < 3, thus Mo({i, n + i}) = {i, n + 2} for all 1 < i < 4. If WoW — ™ + i then 
■u (i + 1) = n + i + 1, since Wo commutes with y?(rj) = i + + n + i + 1). Similarly, if 
Uq{i) = i, then Uq{i + 1) = i+ 1. So, either «o(i) = % and ■Uo^ + z) = n + i for all 1 < % < 4, 
or uo(«) = n + i and ^o(^ + «) — i for all 1 < i < 4. The latter case cannot hold because 
Uq is the disjoint product of two transpositions, thus u (i) = i and u (n + i) = n + i for 
all 1 < i < 4. But, the fact that uq commutes with ipfa) = (i, i + l)(n + i, n + i + 1) for 
all 5 < i < n — 1 also implies that u (i) = i and u (n + i) = n + i for all 5 < i < n. So, 
Uq = Id: a contradiction. 

Case 4: Suppose ip = pn- 

Let sg : Sym 2n — > {±1} denote the signature. Observe that 

(sgo^Xn) = (sgo^)( ( j 1 ) = sg((l,n + 2,n + l,2)) = -1, 

thus sgoy? : _M(£) — > {±1} is surjective. This contradicts the fact that Hi(M(T,)) = {0} 
(see Proposition 3.3). 

Case 5: Suppose ip = Pl2- 

Note that n = 2p + 2 is even, thus 

(sg o ^)( Tl ) = (sg o = sg ^(1, n + 2, n + 1, 2) • J]0', « + i) j = = - 1 , 
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therefore sg o : .M(E) — > {±1} is surjective. As pointed out in the previous case, such 
a homomorphism cannot exist because H-i(M.(T)) = {0}. 

Case 6: Suppose ip = p^ 3 . 

We have 

i— 1 n 

p(Ti) =^(<Ti) = JJ(j> n + j) • (i,n + i + l)(i + l,n + i) • JJ (j,n + j), 

j=l j=i+2 

for all 1 < i < n — \. Set m = <p(tq). Recall that r is conjugate to T4, thus wo is conjugate 
to v?(r 4 ), therefore u is the disjoint product of n transpositions. The permutation u 
commutes with the following three: 

tp(TiMr 2 ) = (l,2,3)(n + l,n + 2,n + 3), 
¥»(ri)^(r 3 ) = (l,2)(3,4)(n + l,n + 2)(n + 3,n + 4), 
V9(r 2 )v9(r 3 ) = (2,3,4)(n + 2,n + 3,n + 4), 

thus 

Uo({l,2,3,n+l,n + 2,n + 3}) = {1, 2, 3, n + 1, ra + 2, n + 3} , 
w ({l,2,3,4,n+ l,n + 2,n + 3,n + 4}) = {1, 2, 3, 4, n + 1, n + 2, n + 3, n + 4} , 
M ({2,3,4,n + 2,n + 3,n + 4}) = {2, 3, 4, n + 2, n + 3, n + 4} , 

hence 

Uo({l,n + l}) = {l,n+l} and w ({4, n + 4}) = {4, n + 4} . 

Moreover, the above two equalities plus the fact that uq commutes with ipijijipij^) = 
(1, 2) (3, 4)(n + 1, n + 2)(n + 3, n + 4) imply that we also have 

M ({2,n + 2}) = {2,n + 2} and w ({3, n + 3}) = {3, n + 3} . 

Similarly, the fact that uq commutes with (pfa) for all 5 < i < n — 1 implies that 
uo({i,n + i}) = {i,n + i} for all 5 < « < n. Finally, since u is the disjoint product of n 
transpositions, we conclude that 

n 
3=1 

Now, a direct calculation shows that 

u ip(r 4 )u = ip(r 4 ) ^ u = (p(T4)u ip(T 4 ) , 
and this contradicts the equality t t 4 t = t 4 t t 4 of Proposition 3.1. □ 
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